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Synapse efficiency diverges due to synaptic pruning following over-growth 
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In the development of the brain, it is known that synapses are pruned following over-growth. This 
pruning following over-growth seems to be a universal phenomenon that occurs in almost all areas 
- visual cortex, motor area, association area, and so on. It has been shown numerically that the 
synapse efficiency is increased by systematic deletion. We discuss the synapse efficiency to evaluate 
the effect of pruning following over-growth, and analytically show that the synapse efficiency diverges 
as 0{\ log c|) at the limit where connecting rate c is extremely small. Under a fixed synapse number 
criterion, the optimal connecting rate, which maximize memory performance, exists. 

PACS numbers: 87.10.-|-e, 89.70,+c, 05.90.-|-m 
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I. INTRODUCTION 

In this paper, we analytically discuss synapse efficiency 
to evaluate effects of pruning following over-growth dur- 
ing brain development, within the framework of auto- 
correlation-type associative memory. 

Because this pruning following over-growth seems to be 
a universal phenomenon that occurs in almost all areas 
- visual cortex, motor area, association area, and so on 
BlllllllllilllilSIIa-we discuss the meaning 
of its function from a universal viewpoint rather than in 
terms of particular properties in each area. Of course, 
to discuss this phenomenon as a universal property of a 
neural network model, we need to choose an appropriate 
model. 

Artificial neural network models are roughly classified 
into two types: feed forward models and recurrent mod- 
els. Various learning rules are applied to the architec- 
tures of these models, and correlation learning corre- 
sponding to the Hebb rule can be considered a proto- 
type of any other learning rules. For instance, correla- 
tion learning can be regarded as a first-order approxi- 
mation of the orthogonal projection matrix, because the 
orthogonal projection matrix can be expanded by cor- 
relation matrices [TJ. In this respect, we can naturally 
regard a correlation-type associative memory model as 
one prototype of the neural network models of the brain. 
For example, Amit et al. discussed the function of the 
column of anterior ventral temporal cortex by means of 
a model based on correlation-type associative memory 
model Also, Sompolinsky discussed the effect of 

dilution. He assumed that the capacity is proportional 
to the number of remaining bonds, and pointed out that 
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a synapse efficiency of diluted network, which is stor- 
age capacity per a connecting rate, is higher than a full 
connected network's one Chechik et al. discussed 

the significance of the function of the pruning following 
over-growth on the basis of a correlation-type associative 
memory model |l5j. They pointed out that a memory 
performance, which is stored pattern number per synapse 
number, is maximized by systematic deletion that cuts 
synapses that are lightly weighted. However, while it is 
qualitatively obvious that synapse efficiency and memory 
performance are increased by a systematic deletion, we 
also need to consider the increase of synapse efficiency 
quantitatively. 

In this paper, we quantitatively compare the effective- 
ness of systematic deletion to that with random deletion 
on the basis of an auto-correlation-type associative mem- 
ory model. In this model, one neuron is connected to 
other neurons with a proportion of c, where c is called 
the connecting rate. Systematic deletion is considered 
as a kind of nonlinear correlation learning [l^ . At the 
limit where the number of neurons N is extremely large, 
it is known that random deletion and nonlinear correla- 
tion learning can be transformed into correlation learning 
with synaptic noise 0, 0| . These two types of deletion, 
systematic and random, are strongly related to multi- 
plicative synaptic noise. First, we investigated the de- 
pendence of storage capacity on multiplicative synaptic 
noise. At the limit where multiplicative synaptic noise 
is extremely large, we show that storage capacity is in- 
versely proportional to the variance of the multiplicative 
synaptic noise. From this result, we analytically derive 
that the synapse efhciency in the case of systematic dele- 
tion diverges as 0(|logc|) at the limit where the con- 
necting rate c is extremely small. We also show that the 
synapse efficiency in the case of systematic deletion be- 
comes 2 1 logc| times as large as that of random deletion. 

In addition to such the fixed neuron number criterion 
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as the synapse efSciency, a fixed synapse number crite- 
rion could be discussed. At the synaptic growth stage, it 
is natural to assume that metabolic energy resources are 
restricted. When metabolic energy resources are limited, 
it is also important that the effect of synaptic pruning 
is discussed under limited synapse number. Under this 
criterion, the optimal connecting rate, which maximize 
memory performance, exists. These optimal connecting 
rates are in agreement with the computer simulation re- 
sults given by Chechik et al . 



II. MODEL 

Sompolinsky discussed the effects of synaptic noise and 
nonlinear synapse by means of the replica method |l4j . 
However, symmetry of the synaptic connections Jij = Jji 
is required in the replica method since the existence of 
the Ljapunov function is necessary. Therefore, there was 
a problem that the symmetry regarding synaptic noise 
had to be assumed in the Sompolinsky theory. To avoid 
this problem, Okada et al. discussed additive synap- 
tic noise, multiplicative synaptic noise, random synap- 
tic deletion, and nonlinear synapse by means of the self- 
consistent signal-to-noise analysis (SCSNA) 0. They 
showed that additive synaptic noise, random synaptic 
deletion, nonlinear synapse can be transformed into mul- 
tiplicative synaptic noise. 

Here, we discuss the synchronous dynamics as, 



N 



h), 



(1) 



where F is the response function, Xi is the output activ- 
ity of neuron i, and —h is the threshold of each neuron. 
Every component in a memorized pattern is an 
independent random variable, 

Prob[C''=±l] = i|^, (2) 

and the generated patterns are called sparse pattern with 
bias a {—1 < a < 1). We have determined that the firing 
rate of states in the retrieval phase is the same for each 
memorized pattern 0,^3- this case, threshold ~h 
can be determined as. 



1 ^ 

a = — ^ sgn(^ J^jXi + h). 

i=l j^i 



(3) 



The firing rate becomes / = (1 -t- a)/2 at the bias a. 

Additive synaptic noise, multiplicative synaptic noise, 
random synaptic deletion, and nonlinear synapse can be 
introduced by synaptic connections in the following man- 
ner. 

In the case of additive synaptic noise, synaptic connec- 
tions are constituted as, 

aN 



Jij — 



J 



iV(l-a2) ^ 

^ ' U— 1 



Y.^S,'t-a){^>;-a) + 5.,, (4) 



where 5ij is the additive synaptic noise. The symmetric 
additive synaptic noise 5ij and 5ji are generated accord- 
ing to the probability. 



5. 



(5) 



where S'j^ is the absolute strength of the additive synaptic 
noise. The parameter Sa is assumed to be 0(1). This 
means that the synaptic connection is 0(1 /Vn). It 
is useful to define the parameter Aa as 

Sa 



A, 



J/(l-a2)' 



(6) 



which measures the relative strength of the noise and 
we call the parameter the variance of the additive 
synaptic noise. Therefore, we define the probability to 
generate the additive synaptic noise 6ij as 



N{0, 



N{l-a^y 



(7) 



In the case of multiplicative synaptic noise, synaptic 
connections are constituted as, 

where is multiplicative synaptic noise. The symmetric 
multiplicative noise and Sji are generated according 
to the probability. 



(9) 



where A\j is the variance of the multiplicative synaptic 
noise. 

In the model of random synaptic deletion, synaptic 
connections are constituted as. 



Jij — 



(10) 



where Cy is a cut coefficient. The synapse that is cut is 
represented by the cut coefficient Cy = 0. In the case of 
symmetric random deletion, the cut coefficients Cy and 
Cji are generated according to the probability. 



Prob[cy = 1] = 1 - Prob[cy = 0] 



cj., (11) 



where c is the connecting rate. 

In the model of nonlinear synapse, synaptic connec- 
tions are constituted as, 



1 



iV(0,l), 



(12) 



(13) 



where p — aN. The nonlinear synapse is introduced by 
applying the nonlinear function f{x) to the conventional 
Hebbian connection matrix Ta. 
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A. Systematic deletion by nonlinear synapse 

Chechik et al. pointed out that a memory perfor- 
mance, which is a storage pattern number per a synapse 
number, is maximized by systematic deletion that cuts 

synapses that are Ughtly weig hted mmiii. Such a 

systematic deletion can be represented by the nonlinear 
function f{x) for a nonlinear synapse. In accordance with 
Chechik et al., we discuss three types of nonlinear func- 
tions (Figs^l^l andOJ. Fig^shows clipped modification 
that is discussed generally as 



fi (z,t) 



FIG. 1: Clipped synapse. 



fiiz,t) 



sgn(z), \z\ > t 
0, otherwise. 



(14) 



Chechik et al. also obtained the nonlinear functions 
shown in Figs|2]ajidJ3|by applying the following optimiza- 
tion principles |15l Il9| . In order to evaluate the effect of 
synaptic pruning on the network's retrieval performance, 
Chechik et al. study its effect on the signal-to-noise ra- 
tio (S/N) of the internal field hi = ^Ylij^i Jij^i + ^ E3- 
The S/N is calculated by analyzing the moments of the 
internal field and was given as 



S/N 



-11 



oc 



E[zf{z)] 
VWW] 



Dz zf{z) 



Dz f(zf 



(15) 



where z has standard normal distribution, i.e., E[z^] — 
V[z] — 1 and Dz is Gaussian measure defined as 



Dz 



dz , z^ \ 



(16) 



and i?, V denote the operators to calculate the expecta- 
tion and the variance for the random variables (i — 
1, ■ ■ ■ , N, n = I,- - ,aN), respectively. The function 
p{f{z),z) denotes the correlation coefficient. Chechik et 
al. considered the piecewise linear function like following 
nonlinear functions. In order to find the best nonlinear 
function /(z), we should maximize p(/(z),z), which is 
invariant to scaling. Namely, the best nonlinear function 
f{x) is obtained by maximizing E[zf{z)] under the con- 
dition that E[f{z)'^] is constant. Let 7 be the Lagrange 
multiplier, it is sufficient to solve 



Dz zf(z) - 7 



Dz f{zY - Co ^ max, (17) 



for some constant cq. Since the synaptic connection be- 
fore acting the nonlinear function obeys a Gaussian 
distribution A^(0, 1), Ea. ljl^l) is averaged over all of the 
synaptic connections. 



fi (z,t) 



FIG. 2: Minimal value deletion. 



Thus, the nonlinear function shown in Fig|21 



^ ' ' 10, otherwise. 



(18) 



is also obtained. The deletion by this nonlinear function 
is called minimal value deletion. Similarly, by adding the 
condition that the total strength of synaptic connection 
J Dz \f{z)\ is constant, the nonlinear function 



z — sgn(z)t, \z\ > t 
0, otherwise 

f2{z,t)^th{z,t), 



(19) 



is obtained. The deletion by this nonlinear function is 
called compressed deletion. We discuss systematic dele- 
tion by using these three types of nonlinear functions 
fi{z,t), f2{z,t) and f:i{z,t) given by Chechik et al (l5| . 



f3 (z.t) 



FIG. 3: Compressed deletion. 
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III. RESULTS 

In this section, the results concerning the multiphca- 
tive synaptic noise, the random deletion, and the nonlin- 
ear synapse are shown, at the limit where the effect of 
these deletions is extremely large. 

The SCSNA starts from the fixed-point equations for 
the dynamics of an A^-neuron network as Eq.QJ. The 
results of the SCSNA for the symmetric additive synaptic 
noise are summarized by the following order-parameter 
equations (see Appendix El : 



Dz<{C-a)Y{z;0 



q 
u 



J Dz<Y{z-if >^, 



Dzz<Y{z-i) >^, 



Y{z;0 



(1 - JC/)2 ^ (1 - a2)2 A 

F[ J(^ - a)m + az 



(20) 
(21) 
(22) 
(23) 



aJ^U 



J2 



1- JU (1 - a 



Y{z;0]m 



where < • • ■ >^ implies averaging over the target pattern, 
m is the overlap between the 1st memory pattern and 
the equilibrium state x is defined as 



1 



N 



N(l~a^) ^ 



(25) 



note that generality is kept even if the overlap was defined 
by only the 1st memory pattern, q is Edwards- Anderson 
order-parameter, [/ is a kind of the susceptibility, which 
measures sensitivity of neuron output with respect to the 
external input, Y(z\^ is effective response function, 
is the variance of the noise. We set the output function 
F(x) = sgn(x)— a in the following sections, where domain 
of the X variable is F(x) = 1 — a when a: < 0, otherwise 
F{x) = -1 - a. 

According to Okada et al. , the symmetric additive 
synaptic noise, the symmetric random deletion, and the 
nonlinear synapse can be transformed into the symmetric 
multiplicative synaptic noise as follows (see Appendix^ : 
the additive synaptic noise is 



A2 



,2\2 ■ 



the random deletion is 



A2 



and the nonlinear synapse is 



A^ 



J2 



(26) 



(27) 



(28) 
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FIG. 4: Overlaps in the random deletion network. The curves 
represent the theoritical results. The dots represent simula- 
tion results with A'" = 3000 and / = 0.1 for the connecting 
rate c = 0.1, c = 0.3, and c = 1.0. 



where J, are 



J = J Dz zf{z), 

p = fDzfizr. 



(29) 
(30) 



In the following sections, symmetries of the additive 
synaptic noise, the multiplicative synaptic noise and ran- 
dom deletion are assumed.® Storage capacity can be ob- 
tained by solving the order-parameter equations. 

Figure 0] shows m{a) curves in the random deletion 
network with the number of neurons A^ = 3000 and the 
firing rate / = 0.1 for the connecting rate c — 0.1, c = 0.3, 
and c = 1.0. It can be confirmed that the theoretical 
results of the SCSNA are in good agreement with the 
computer simulation results from Fig^J Since it is known 
that theoretical results obtained by means of the SCSNA 
are generally in good agreement with the results obtained 
through the computer simulations using various models 
that include synaptic noise, we treat the results by means 
of the SCSNA only [S |2fl HI HI HI H El lH . 

Through the relationships of Eas. l|2()|) - (|30|l . the sym- 
metric additive synaptic noise, the symmetric random 
deletion, and the nonlinear synapse can be discussed 
in terms of the symmetric multiplicative synaptic noise. 
Therefore, first of all, we deal with the multiplicative 
synaptic noise. 



A. Multiplicative synaptic noise 

Figure O shows the dependence of storage capacity on 
the multiplicative synaptic noise. As it is clear from 
Figl^l storage capacity ac is inversely proportional to the 
variance of the multiplicative synaptic noise A^j, when 
the multiplicative synaptic noise is extremely large. Stor- 
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FIG. 5: Dependence of storage capacity Oc on the multiplica- 
tive synaptic noise A%j at the firing rate / — 0.5. Comparison 
of asymptote and the results from the SCSNA. 
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FIG. 6: Dependence of storage capacity on the multiplica- 
tive synaptic noise A|,^. It can be confirmed that the order 
of the asymptote does not depend on the firing rate. 



age capacity Uc asymptotically approaches 

2 



(31) 



(see Appendix [HJ . In the sparse limit where the firing 
rate is extremely small, it is known that storage capacity 
becomes ~ l/(/| log/|) [11 S IH IH 113 ■ 

Figure |S1 shows the results from the SCSNA and the 
asymptote at the firing rate / = 0.5. Figure shows the 
results from the SCSNA at various firing rates. It can be 
confirmed that the order of the asymptote O(-r^) does 
not depend on the firing rate from Fig|6l 



FIG. 7: Dependence of the synapse efficiency S^ff on the 
random deletion with the connecting rate c at the firing rate 
/ = 0.5. Comparison of asymptote and the results from the 
SCSNA. 
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FIG. 8: Dependence of the synapse efficiency Seff on the 
random deletion with the connecting rate c at various firing 
rates. It can be confirmed that the order of the asymptote 
does not depend on the firing rate. 



according to the asymptote of the multiplicative synap- 
tic noise in Ea. H31|l . In the random deletion, the synapse 
efficiency g^ff, which is storage capacity per the con- 
necting rate [l^ lla |. i.e., storage capacity per the input 
of one neuron, and defined as 



Seff = 



approaches a constant value as 



(33) 



B. Random deletion 

Next, we discuss the asymptote of the random dele- 
tion. The random deletion with the connecting rate c can 
be transformed into the multiplicative synaptic noise by 
Ea. (|27|l . Hence, at the limit where the connecting rate c 
is extremely small, storage capacity becomes 



2c 



7r(l - c) 



(32) 



(34) 



according to Ea. H32ll at the limit where the connecting 
rate c is extremely small. 

Figure [7\ shows the result from the SCSNA and the 
asymptote at the firing rate / — 0.5. Figure |H1 shows 
the results from the SCSNA at various firing rates. It 
can be confirmed that the order of the asymptote, 0(1) 
with respect to c, does not depend on the firing rate from 
FigH 
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C. Systematic Deletion 

1. Clipped synapse 



Synapses within the range —t<z<t are pruned by 
the nonhnear function of Eq. 1)14(1 . 

The connection rate c of the synaptic deletion in 
Eg. lfT^ is given by, 



Dz 



{z\h(z,t)^0] 



^t-iex,j(-^), i 



oo, 



(35) 



since the synaptic connection Ty before acting the non- 
hnear function of Eq. p3(l obeys the Gaussian distribution 
N{0, 1). Next, J, J of Eqs.lEni and ^ become 



J — 2 J Dz zsgn(z) 
tc, t 



-exp(--) 



-> CXD, 

Dz = 1 



(36) 
(37) 



Hence, the equivalent multiplicative synaptic noise A^^ 
is obtained as, 



- — 



oo. 



(38) 



The relationship of the pruning range t and the connect- 
ing rate c 

i2 = -21ogc, (39) 

is obtained by taking the logarithm of Ea. (|35|l at t ^ 
oo limit. Therefore, at the limit where the equivalent 
connecting rate c is extremely small, storage capacity ac 
can be obtained 



ac = clogc, 



(40) 



through Eas. pifl . (|38|l . and H39|l . The synapse efficiency 
becomes 



^eff - — 



4, 

- — logc. 

TT 



(41) 



Figure shows the results from the SCSNA and the 
asymptote at the firing rate / = 0.5. FigurelTUIshows the 
results from the SCSNA at various firing rates. It can be 
confirmed that the order of the asymptote O(logc) does 
not depend on the firing rate from Fig llOl 
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FIG. 9: Dependence of the synapse efficiency S^ff with the 
clipped synapse on the connecting rate c at / = 0.5. Com- 
parison of the results from the SCSNA and asymptote. 
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FIG. 10: Dependence of the synapse efficiency S^ff with the 
clipped synapse on the firing rate /. It can be confirmed that 
the order of the asymptote does not depend on the firing rate. 



where the connecting rate c and J, J of Eas. H29|l and H30f) 

are 

t 



Dz = 1 - erf(^) 

{z\f2{z,t)^0} V2 



lexp(-y), t 



oo, 



J 



r = J, 



-texp(--)-fl-erf(-^) 



--<exp(- — ), t 



oo. 



(43) 



(44) 
(45) 



2. Minimal value deletion 



respectively. Hence, at the limit where the equivalent 
connecting rate c is extremely small, storage capacity Oic 
and the synapse efficiency Sef / can be obtained through 
Eqs.jnUlC 1121), and jSni as follows. 



In a similar way, the equivalent multiplicative synap- 
tic noise A|j of the systematic deletion of Eq. lfTH|l is ob- 
tained as follows. 



J2 



1 



1 



t 



OO. 



(42) 



Oic = clogc, 

TT 



(46) 
(47) 



respectively. Figure shows the results from the SC- 
SNA and the asymptote at the firing rate / = 0.5. Fig- 
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FIG. 11: Dependence of the synapse efficiency Se// with 
the minimal value deletion on the connecting rate c at time 
/ — 0.5. Comparison of the results from the SCSNA and the 
asymptote. 
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FIG. 12: Dependence of the synapse efficiency Seff with 
the minimal value deletion on the firing rate /. It can be 
confirming that the order of the asymptote does not depend 
on the firing rate. 



Dz^ l-erf(^), 

{z\h(z,t)^Q} V2 

2" 

_t-icxp(--), i^oo, (49) 
c (50) 

/OO 
-OO 

rOO 

~2t Dz Mz,t)h{z,t) 



^^exp(--) + l-erf(^) 



2c 



, t — > OO, 



- 2t 



-exp(~-) 



(51) 



respectively. Here, we use a asymptotic expansion equa- 
tion of the error function 



erf(a;) = 1 



\x 2x^ Ax^ 



for a; ^ 1. In order for the first term and the second 
term of Ea. H51|l to be same order, the asymptotic ex- 
pansion equation has taken the approximation of 0{t^^) 
and 0{t~^) respectively. The equivalent multiplicative 
synaptic noise in the case of systematic deletion becomes 
double that of the clipped synapse of Ea. (|38|l and the 
minimal value deletion of Ea. (|48|l . Therefore, at the limit 
where the equivalent connecting rate c is extremely small, 
storage capacity ac and the synapse efficiency Seff can 
be obtained through Eqs.(|3T}C (^J), and as follows, 



urell2lshows the results from the SCSNA at various firing 
rates. It can be confirmed that the order of the asymp- 
tote does not depend on the firing rate from Figll2l 



otc = clogc, 

TT 

2 

Seff = logC, 



(53) 
(54) 



3. Compressed deletion 



Again, in the similar way. 

The equivalent multiplicative synaptic noise A^,^ of the 
systematic deletion of Eq. H19|) is given by 



J2 2 



(48) 



where the connecting rate c and J, J of Eas. H29|l and H3U|) 



respectively. Figure El shows the results from the SC- 
SNA and the asymptote at the firing rate / = 0.5. Fig- 
ure El shows the results by the SCSNA at various firing 
rates. It can be confirmed that the order of the asymptote 
0(log c) does not depend on the firing rate from Figll4l 
Figure El shows the dependence of the synapse effi- 
ciency Sef f on the connecting rate c obtained by means 
of the SCSNA. Table ^ shows the asymptote of storage 
capacity with the random deletion and the systematic 
deletion. Hence, when using minimal value deletion as 
the simplest from of systematic deletion we found that 
the synapse efficiency in the case of systematic deletion 
becomes 



Sef f (systematic deletion) ^ | log c| 
Sef f (random deletion) — 



2|logc|, (55) 
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TABLE 11: Asymptote of the synapse efficiency by the ran- 
dom deletion and the systematic deletion at the firing rate 
/ = 0.5. 



Types of deletion 


Synapse efficiency 




(Asymptote) 


random deletion 




systematic deletion clipped synapse 


(4A)|logc| 


minimal value deletion 


(4/^)1 log c| 


compressed deletion 


(2/7r)ilogcj 



FIG. 13: Dependence of the synapse efficiency Seff with 
the compressed deletion on the connecting rate c at / = 0.5. 
Comparison of the results from the SCSNA and the asymp- 
tote. 
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FIG. 14: Dependence of the synapse efficiency Seff with the 
compressed deletion on the firing rate /. It can be confirming 
that the order of the asymptote does not depend on the firing 
rate. 
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FIG. 15: Comparison of the synapse efficiency with the ran- 
dom deletion and that with the systematic deletion at the 
firing rate / — 0.5. 



IV. THE MEMORY PERFORMANCE UNDER 
LIMITED METABOLIC ENERGY RESOURCES 



thus we have shown analytically that the synapse ef- 
ficiency in the case of systematic deletion diverges as 
0(|logc|) at the limit where the connecting rate c is 
extremely small, and have shown that the synapse ef- 
ficiency in the case of the systematic deletion becomes 
2 1 logc| times as large as that of the random deletion. 



TABLE I: Asymptote of storage capacity by the random 
deletion and by the systematic deletion at the firing rate / = 
0.5. 



Types of deletion 


Storage capacity 




(Asymptote) 


random deletion 


{2/7v)c 


systematic deletion clipped synapse 


(4/7r)c| log c| 


minimal value deletion 


(4/7r)c log c 


compressed deletion 


(2/7r)c|logc| 



Until the previous section, we have discussed the effect 
of synaptic pruning by evaluating the synapse efficiency 
which is the memory capacity normalized by connect- 
ing rate c. When the connecting rate is c, the synapse 
number per one neuron decreases to cN. Therefore, the 
synapse efficiency means the capacity per the input of 
one neuron. In the discussion by the synapse efficiency, 
the synapse number decreases when the connecting rate 
is small. 

In addition to such the fixed neuron number crite- 
rion, a fixed synapse number criterion could be discussed. 
At the synaptic growth stage, it is natural to assume 
that metabolic energy resources are restricted. When 
metabolic energy resources are limited, it is also impor- 
tant that the effect of synaptic pruning is discussed un- 
der limited synapse number. Chechik et al. discussed 
the memorized pattern number per one synapse under a 
fixed synapse number criterion [l5jD They pointed out 
the existence of an optimal connecting rate under the 
fixed synapse number criterion and suggested an expla- 
nation of synaptic pruning as follows: synaptic pruning 
following over-growth can improve performance of a net- 
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work with limited synaptic resources. 

The synapse number is N"^ in the fuU connected case 
c — 1. We consider the larger network with M(> N) 
neurons. The synapse number in the lager networks with 
the connecting rate c becomes cM^ . We can introduce 
the fixed synapse number criterion by considering a larger 
network which has M neurons, i.e., 



iV2 ^ cAf^ 



(56) 



synapses at the connecting rate c. The memorized pat- 
tern number per one synapse becomes 



Pc 



Pc 



TV 



cM2 7V2 ^ 
where the critical memorized pattern number pc is 



(57) 



(58) 



We define the coefficient Uc/ \/c as memory performance. 
We discuss the effect of synaptic pruning by the memory 
performance. Under limited metabolic energy resources, 
the optimal strategy is the maximization of the memory 
performance. Chechik et al. showed the existence of the 
optimal connecting rate which maximize memory perfor- 
mance 15]. The memory performance can be calculated 
by normalising the capacity, which is given by solving the 
order-parameter equations, with y^. FigureElshows the 
dependence of the memory performance on the connect- 
ing rate in three types of pruning. It is confirmed that 
there are the optimal values which maximized the mem- 
ory performance by each deletions. The optimal connect- 
ing rates of clipped synapse, minimal value deletion, and 
compressed deletion are c = 0.036,0.038,0.084, respec- 
tively. This interesting fact may imply that the memory 
performance is improved without heavy pruning. These 
optimal values agree with the computer simulation re- 
sults given by Chechik et al 



clipped synapse - 
minimal value deletion 
compessed deletion - 




0.4 0.6 
connecting rate c 



FIG. 16: Memory performance Qfc/\/c of networks with difler- 
ent number of neurons but the same total number of synapses 
as a function of the connecting rate c in the case of clipped 
synapse (the solid line), minimal value deletion (the dotted 
line), and compressed deletion (the dashed line). 



V. CONCLUSION 

We have analytically discussed the synapse efficiency, 
which we regarded as the auto-correlation-type associa- 
tive memory, to evaluate the effect of the pruning fol- 
lowing over-growth. Although Chechik et al. pointed 
out that the synapse efficiency is increased by the sys- 
tematic deletion, this is qualitatively obvious and the in- 
crease in the synapse efficiency should also be discussed 
quantitatively. At the limit where the multiplicative 
synaptic noise is extremely large, storage capacity ac is 
inversely as the variance of the multiplicative synaptic 
noise A^^. From this result, we analytically obtained 
that the synapse efficiency in the case of the systematic 
deletion diverges as 0(| logc|) at the limit where the con- 
necting rate c is extremely small. 

On the other hand, it is natural to assume that 
metabolic energy resources are restricted at the synap- 
tic growth stage. When metabolic energy resources are 
limited, i.e., synapse number is limited, the optimal con- 
necting rate, which maximize memory performance, ex- 
ists. These optimal values are in agreement with the 
results given by Chechik et al p^ . 

In the correlation learning, which can be considered a 
prototype of any other learning rules, various properties 
can be analyzed quantitatively. The asymptote of 
synapse efficiency in the model with another learning 
rule can be discussed in a similar way. As our future 
work, we plan to further discuss these properties while 
taking into account various considerations regarding 
related physiological knowledge. 
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APPENDIX A: SCSNA FOR ADDITIVE 
SYNAPTIC NOISE 

Derivations of the order-parameter Eas. (|20|l - H24|) are 
given here. The SCSNA starts from the fixed-point equa- 
tions for the dynamics of the iV-neuron network shown 
as Eq.(^|. The random memory patterns are generated 
according to the probability distribution of Eq. Q . The 
syanaptic connections are given by Eq.Q). The asym- 
metric additive synaptic noise, 5ij and 5ji are indepen- 
dently generated according to the probability distribu- 
tion of Eq.Q. Moreover, we can analyze a more general 
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case, where Sij and Sji have an arbitary correlation such 
that 



(Al) 



In this general case, the symmetric and the asymmetric 
additive synaptic noise correspond to fc^ = 1 and fc^ = 0, 
respectively. Here, we assume the probability distribu- 
tion of the additive syanptic noise is normal distribution 
Sij ^ N{0, -|^^j4^^A^). However, any probability dis- 
tributions, which have same average and variance, can be 
discussed by the central limit theorem in the similar way 
of the following disscussion. Defining the loading rate as 
a — p/N, we can write the local field hi for neuron i as 

N 

hi = JijXj 

aN N 

= J^{£,i - a)m^ + ^dijXj - Jaxi, (A2) 

where is the overlap between the stored pattern 
and the equilibrium state x defined by 



1 " 



(A3) 



The second term including Xj = F{hj + h) in Ea. ljA2|) 
depends on 5ji . The 5ij dependences of Xj are extracted 
from Xj , 



where 



Xj — F{hj SjiXi^., 



F'{hj - SjiXt). 



(A4) 

(A5) 
(A6) 



Substituting Eq. (|A4|) into Eq. HA2p , the local field /ij can 
be expressed as 

aN N 

hi = J ^(Cf - a)m^ + ^ Sijxf'"^ ~ Jaxi 

N 



~\~Xi ^ ^ SijSjiX j 



(A7) 



We assume that Ea.HA7|l and Xi = F{hi + h) can be 
solved by using the effective response function F{u) as. 



JV 



f[jY.{S,1 ~a)m^ + Y.^,,x] 



^=1 



(A8) 



Let be the target pattern to be retrieved. Therefore, 
we can assume that toi = 0(1) and = 0{l/^/N), fi > 



1. Then we can use the Taylor expansion to obtain 



JV 



1 



N 



7V(l_a2)(l_ J[/)^ 



5](er-aK(^\ (A9) 



by substituting Ea. (|A8|l into the overlap defined by 
Ea. HA3|l . where 



aN 



N 



aN 



N 



p' ( ^ E(^''^ - «)™- + E -^^^4 ) (All) 



1 ^ 



(A12) 



Equations (jA7|) and IjAOp give the following expression 
for the local field: 



1- JC/+^''(l-a2)2"^ 



rA 



Uxi 



J 



N aN 



N{1- a'^){l- JU) 



EE(^r-«)(^^-«)4'' 



j^i Ai=2 



N 



^5ijX 



(A13) 



Note that the second term in Ea. (|A13|l denotes the ef- 
fective self-coupling term. The third and the last terms 
are summations of uncorrelated random variables, with 
mean and variance. 



J2 



A2(l-a2)2(l- j;7)2 

N aN 

xEE(^f-«)'(^.-«)'(4'^)' 



j^t M=2 



aj2 



[i-juy 



E4(-f")^- 



(l-«2): 



(A14) 



rA^g, (A15) 
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respectively. The cross term of these terms have van- 
ished. Thus, we finaUy obtain 



l^i — J{^i — a)mi + azi 



1 - JU 



[l-JUf (l-a2)2 



Ux^ (A16) 
(A17) 



from Eas. (|A14|l and HA15|I . where ~ 7V(0, 1). Equation 
is given by Ea. (|A17|l . Finally, after rewriting i^^^ ^, 
TOi m, Zi ^ z, and cci ^ the results of the 

SCSNA for the additive synaptic noise are summarized 
by the order-parameter equations of Eas. (|20() - (|22|l as, 

m = j Dz<{i- a)Y{z; ^ 

q = J Dz<Y{z-£,f >5, 

U = ^ J Dzz< Y{z;^) >5, 

where the effective response function Y{z;^) becomes 

Y{z;^) = F(j{^~a)m + (7z + h 



A:.-- — A2 



1 - Jt/ (1 - a2)2 -4 



c/r(z;0 

(A18) 



The effective response function of Ea. H24|) can be ob- 
tained by substituting ks — 1 into Eg. lAlSj) . 



APPENDIX B: EQUIVALENCE AMONG THREE 
TYPES OF NOISE 



kg, = 0, respectively. Here, we assume the probability 
distribution of the multiplicative synaptic noise is nor- 
mal distribution ~ N{0,A\,j). The local field hi for 
neuron i becomes 



aN N 

N(l-a^) ^ 

-OLXi . 



aN N 



(B2) 



where is the overlap defined by Eq. ljA3(l . The second 
term including Xj — F{hj + h) in Ea. (|B2|l depends on 
Eji. The £ij dependences and ^j" dependences of X j are 
extracted from Xj, 



Xj = _|_ ^{M,eji}^/(M)(ej>) 



(B3) 



(B4) 



where 



X 
X 



(M)feO ^ (B5) 
' = F'ihj^h]''-'''^). (B6) 



The multiplicative synaptic noise, the random synaptic 
deletion, and the nonlinear synapse can be discussed in 
the similar manner to Appendix IXI 

1. Multiplicative synaptic noise 

Derivations of the equivalent noise Ea. (|26|l is given 
here. We can also analyze by a similar manner to the 
analysis of the additive synaptic noise. The syanaptic 
connections are given by Eq.(|SJ|. The asymmetric multi- 
plicative synaptic noise, and Eji are independently 
generated according to the probability distribution of 
Eq.Q. We analyze a more general case, where Sij and 
dji have an arbitary correlation such that 

Cov[eij,ej,] = k,Ali, -l<k,<l. (Bl) 

In this general case, the symmetric and the asymmetric 
multiplicative synaptic noise correspond to /cg = 1 and 



We assume that Ea. l)B2|l and Xi = F{hi+h) can be solved 
by using the effective response function F(u) as, 

p 

tj.=i 

aN N 

(B7) 

Let be the target pattern. We substitute Eq. I|B7|I into 
the overlap defined by Eq. ljA3|l and expand the resultant 
expression by {^^ — a)m^ > l)j which has the order 
oiO{l/^/N). This leads to 
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where 



aN N 



(B9) 



and U is defined by the similar way of Ea. (jA12|l in the 
case of the additive synaptic noise. Equations I^JJilESJ 
and l)B8|l give 



hi — {^l ~ a)m\ + a 
1 



1 



+ 



1 - U 

N aN 



j 



aN N 



(BIO) 



The third and last terms can be regarded as the noise 
terms. The variance of the noise terms becomes 



aq 



{l-Uf 



aAljq 



(Bll) 



Thus, after rewriting ~^ ^ and mi —> m, we obtain the 
effective response function: 



= F{{^-a)m + az + h 



1 - U 



UY{z;0 



(B12) 



Finally, the equivalence between the multiplicative 
synaptic noise and the additive synaptic noise is obtained 
as follows, 



J - 1, 
A\ = ail - a^fAl, 



(B13) 
(B14) 
(B15) 



by comparing Eas. HBll|) and (|B12|) to Eas. ljAlTp and 



2. Random deletion 

Derivations of the equivalent noise Ea. (|27|l is given 
here. The random deletion has similar effects to the mul- 
tiplicative synaptic noise. Therefore, we analyse by a 
similar way to the analysis of the multiplicative synaptic 
noise. The syanaptic connections are given by Ea. (|10|l . 



The asymmetric cut coefficients are independently gener- 
ated according to the probability distribution of Ea. (fTT|) . 
We analyze a more general case, where Cij and Cji have 
an arbitary correlation such that 

Cov[cy , Cjj] = kcVai[cij], -1 < fee < 1 (B16) 
Var[c.,] = E[(c.,)'] - (E[cy])2 

= c(l-c). (B17) 

In this general case, the symmetric and asymmetric ran- 
dom deletion correspond to fee = 1 and kc — 0, respec- 
tively. According to a similar analysis of the multiplica- 
tive synaptic noise, the local field becomes 



hi = {^l — a)mi + a 

] 

^ N{l-a^){l-U) 



1 fcc(l-c) 
1-U c 



U Xi 



N aN 
^ aN N 

7Vc(l-a2)EE(^*J-^) 



where 



, aN 



(B18) 
(B19) 



aN N s 

EE(^^^--^)(^r-«)(cj-«)^r''"7 



(B20) 



Nc{l - a2) 
^{cjk - c)(C^^ - a)(^^ - a)xk 



N 



Cji C 

'iVc(l-a2) 



Cji C 



-(er-a)(er-a> 



(B21) 



iVc(l - a2) 

and U is defined by Ea. (|A12p similarly. The variance of 
the noise term is given by 



aq 



1 - c 



(1 - Ur ' - c - (^22) 

Thus, after rewriting —> ^ and mi m, the effective 
response function becomes 



= F(^{^-a)m + az + h 
1 , k,{l-c) 



+a 



1 - U 



UY{z;0]. 

(B23) 
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Finally, the equivalence between random deletion and the 
additive synaptic noise is obtained as follows, 



J 

kg 



1, 



2\2 - 



a(l - a^) 



(B24) 
(B25) 
(B26) 



by comparing Eas. HB22|) and (|B23|) to Eas. ljAiyp and 
(|A18() . Substituting Ea. (|B25|l into Eg. p 14(1 . we obtain 
the equivalence of Eq. (|27fl . 



3. Nonlinear synapse 

Derivations of the equivalent noise Ea. ((28|l is given 
here. The effect of the nonlinear synapse can be sepa- 
rated into a signal part and a noise part. The noise part 
can be regarded as the additive synaptic noise. 

The systematic deletion of synaptic connections can be 
achieved by introducing synaptic noise with an appropri- 
ate nonlinear function f{x) Note that Tij obeys 
the normal distribution iV(0, 1) for p = aN oo. Ac- 
cording to this naive S/N analysis [16| . we can write the 
connections as 



VP 

N 



J 



N{1 - a? 



J 



N{l~o?) 



E(c"-«)(c;-a) 



VP 

N 



{JT,,-[f{T,,)~m,]}. 



(B27) 



The following derivation suggests that the residual over- 
lap for the first term in Ea. ((B27|l is enhanced by a 
factor of 1/(1 — JJ7), while any enhancement to the last 
part is canceled because of the subtraction. It also im- 
plies that the last part corresponds to the synaptic noise. 
For the SCSNA of the nonlinear synapse, we can ana- 
lyze by a similar manner of the analysis of the additive 
synaptic noise. We obtain the local field: 



hi = J{^1 — a)mi + a 



1 - JU 



+ (J2 - J2) 



Uxi 



p N 



(B28) 



where 



p 

N 



F[h,~^[f{T,,)-JT,,]xi)], (B30) 



Vp(l-a2) 



Y.i^'^-aWj-a), (B31) 



and U is defined by Ea. (|A12|l similarly. The variance of 
the noise term is given by 



2 aJ-'q , ~2 2n 



(1- JC/)2 



(B32) 



Thus, after rewriting ^ — s- ^ and m\ — > to, The effective 
response function becomes 



J2 



1 - JU 



+ (J2 - J') 



(B33) 



Finally, the equivalence between the nonlinear synapse 
and the additive synaptic noise is obtained as follows. 



J = j Dx xf{x) 
J2 = / Dxf{xf 



(B34) 
(B35) 
(B36) 



by comparing Eas. HB32|) and (|B33|) to Eas. ljAlTp and 
(|Il8)l . Substituting Ea. l|B34jl into Ea. (|BT4|) . we obtain 
the equivalence of Eq. H28() . 



APPENDIX C: ASYMPTOTE FOR LARGE 
MULTIPLICATIVE SYNAPTIC NOISE 

Derivations of the asymptote of storage capacity in a 
large multiplicative synaptic noise Am is given here. 

In Eqs.ll^ let a = 0, J = 1, and F{x) = sgn(a;), 

the order-parameter equations become 



erf 



/ TO 

VV2^ 



= 1, 



U = 



1 2 
-W-cxp 

(T V TT 



m 
2^ 



(CI) 
(C2) 

(C3) 



14 



the threshold becomes h = 0, the effective response func- 
tion of Eq. 1241) and the variance of the noise become 



Y{z;£_) = sgn (^m + crz) 
o a 



a 



A2 



(C4) 
(C5) 



[l-Ur 

respectively, where the error function erf(a;) is defined as 



erf(j;) = -= / e"" du. (C6) 
VT^ Jo 

The slope of the r.h.s. of Ea. l|Cip is given by 



am 



Equation ljUl|l has nontrivial solutions m ^ within the 
range where the slope of the r.h.s. of Ea. HC7l) at m = is 
greater than 1. Therefore, the critical value of the noise 
is given by 



2/7r. 



(C8) 



This shows that a retrieval phase exists only for cr < CTc 
We define the parameter r(< 1) defined as 



(T 



(C9) 



to solve for m as a function of a in the vicinity of this 
critical value tXc- The critical value of the additive synap- 
tic noise is discussed in the case of r ~ 1 . The overlap m 
shows the first order phase transition when is small, 
but it is regarded as the second order phase transition 
at large Aa region. The overlap becomes m <^ I when 
T ~ 1 and Aa is sufficiently large, therefore the nontrivial 
solution of m is given as 



m m 



+ 0(m'*) = crcTy/6{l-T), (CIO) 



by Taylor expansion including terms up to the third or- 
der. Substituting Ea. HC10|) into Ea. (|C3p . U becomes 

U~^l-^^+Oim^) = 3-2T-\ (Cll) 



From Eq.|5BJ, the variance of the multiplicative synap- 
tic noise A^ is related to the variance of the additive 
synaptic noise A\ as 



(C12) 



when bias a — 0. Therefore, substituting Eas. ljClip and 
(ICTl into Eq.(|nni), the variance of the noise a is given 
as 



ar 



4(1 -T 

The loading rate a becomes 



aA 



M- 



a 



4Ai,(l 



(C13) 



(C14) 



When the overlap is small enough, i.e., m <ti 1, the order- 
parameter equations of Eqs. (|C1I) - (|C5|I reduce to Eq. 
(IC14|I . Solving Ea. HC14p for the fixed value of a and 
Am, we obtain the parameter r. Substituting r into Eq. 
(|(nop , we can obtain the overlap m for given a and Aj\/ . 
It is easily confirmed that the r increases with a for the 
fixed value of Am- This means that the maximal value 
of T which holds Eq. (jC14p corresponds to the maximum 
value of a, that is storage capacity ac- The critical value 
Tf. is equal to the value which maximizes the loading rate 
of Eq. (|C14|I and becomes 



(2Am) 



2/3 



1 + (2Am)2/3 



1-(2Am)-^/^ 



(C15) 



in a large Am limit. Therefore, substituting Eq. I|C15|I 
into Ea. (|C14p . we obtain Ea. (|^ as follows: 



(C16) 
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